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ABSTRACT 



A survey of 32 Japanese and 44 native English-speaking 
teachers of English as a Second Language investigated how the two groups 
evaluate the English speech skills of Japanese students. A 59-item 
questionnaire was designed to elicit comparative information on definition of 
oral proficiency, criteria (including newer ones derived from instruction 
focusing on communicative competence) used to assess oral skills, and the 
relative importance attached to these criteria. Results suggest significant 
differences overall between Japanese and native English-speakers ' standards 
in two main assessment categories, fluency and discourse factors, although no 
significant differences appeared within subcategories of these criterion 
groups. The questionnaire is appended. (MSE) 
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INTRODUCTION 

Since the emergence of the notion of com- 
municative competence, the role of oral profi- 
ciency has become more central in language 
teaching than it was in the era of structuralism. 
In response to this phenomenon, tests of speak- 
ing ability have also changed. In such tests the 
crucial element is the rating of students’ per- 
formance. A few problems arise, however, 
when testing English speaking ability. First, 
the definition of speaking ability may not be 
clearly established. Second, the meaning of 
categories used to rate speaking ability, includ- 
ing newer ones derived from communicative 
competence, may not be understood in the 
same way by both Japanese English teachers 
and native English teachers. Third, these crite- 
ria may not be considered equally important by 
both groups of teachers in evaluating the En- 
glish speaking ability of Japanese students. 

A primary goal of foreign language teaching 
is to enable students to communicate with 
native speakers. Thus, in foreign language 
tests, especially speaking tests, native speaker 
evaluation standards are crucial when rating 
categories are decided. Following is a look at 
how Japanese English teachers and native En- 
glish teachers evaluate the speaking ability of 
Japanese students. 

HYPOTHESES 

1. In some criteria there may be significant 
differences in the rating standards of native 
and Japanese English teachers. 

2. When there are no significant differences 
between the two groups of evaluators, some 
criteria will be rated higher by both Japanese 
and native teachers while both groups of teach- 
ers will give lower points to other criteria. 



SUBJECTS 

Seventy-six college English teachers-32 
Japanese English teachers and 44 native En- 
glish teachers-were chosen as subjects. All 
subjects had been teaching English at the col- 
lege level for at least two years. 

RESEARCH INSTRUMENT 

A 59-item questionnaire (see Appendix) was 
used as the research instrument. All items were 
rated on a 1-5 scale (l=not important, 5=im- 
portant). These 59 items consisted of 1 1 main 
categories and 48 sub-categories which were 
selected mainly from the following four 
sources: informal interviews with the subjects; 
Richards’ (1990) notion of conversation strat- 
egies; Nakamura’s (1990) previous work with 
native speaker evaluation points; Bachman’s 
(1990) linguistic theory of Communicative 
Language Ability. 

To arrive at the 59 items, pilot tests were 
conducted with three groups: native English 
teachers, Japanese English teachers, and na- 
tive speakers who were not teachers. This was 
to ensure that in the final questionnaire, all 
subjects would clearly understand the mean- 
ing of each item. 

PROCEDURE 

1. Two hundred college English teachers at 
four conferences were asked to answer and 
mail in questionnaires. A total of 76 completed 
questionnaires were returned. 

2. The mean score and the standard deviation 
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of each group (Japanese English teachers and 
native English teachers) were computed. T- 
tests were conducted to elicit any significant 
differences between the mean scores of both 
groups. 



order of importance of some categories (see 
Mean). Native English teachers ranked Flu- 
ency the most important, followed by Dis- 
course factors and Content. In contrast, Japa- 
nese teachers ranked Content as the most im- 



FINDINGS 

Table 1 : 

Two main categories, Fluency and Discourse 
factors show significant differences in the mean 
scores between the two groups. The Japanese 
and native English teachers also differed on the 



portant category, followed by Pronunciation 
( suprasegmental ), and Vocabulary use. Flu- 
ency was ranked as least important. 

Thus, in the 1 1 main categories, there are 
differences in the order of importance of cat- 
egories as well as in the differences of the mean 
scores between the two subject groups. 



Table 1 






Mean 


SD 




t 




Japanese 


Native 


Japanese 


Native 




1. Grammatical accuracy 


3.09 


3.09 


.73 


.80 


.02 


2. Vocabulary use 


3.53 


3.73 


.80 


.66 


1.17 


3. Pronunciation (segmental features) 


3.44 


3.11 


.95 


.95 


1.47 


4. Pronunciation (suprasegmental features) 


3.59 


3.30 


.95 


.82 


1.46 


5. Fluency 


2.87 


3.97 


.69 


.84 


5.65”* 


6. Discourse factors (cohesion and coherence) 


3.25 


3.89 


.72 


.84 


3.46*** 


7. Content 


3.84 


3.82 


1.02 


1.00 


.11 


8. Level of speaker's confidence 


3.34 


3.43 


1.13 


1.07 


.35 


9. Sociolinguistic competence 


3.28 


3.36 


.85 


.84 


.42 


10. Strategic competence 


3.34 


3.68 


.94 


.93 


1.56 


11. Illocutionary competence 


3.19 


3.43 


.82 


1.02 


1.12 




*** p < .001, two-tailed 








N.B. Japanese Teachers (n= 


=32), Native Teachers (i 


i=44) 






Table 2: 




Frequency of uncompleted sentences and Cor- 


There are no significant differences in the 


rect speed of speech, generally regarded as an 


Fluency sub-categories in T able 2, even 


ithough 


important factor of “fluency, 


” are rated as less 


Fluency , as one of the main categories, had 


important by both groups of teachers, while 


significant differences within the scope of the 


Ease of speaking is rated as highly important 


1 1 main categories. Table 2 also shows that 


by both groups. 






Table 2 






Mean 


SD 




t 


Fluency 


Japanese 


Native 


Japanese 


Native 




T Proper use of pauses 


3.25 


3.14 


.76 


.88 


.59 


2. Frequency of uncompleted sentences 


2.94 


2.86 


.72 


1.09 


.33 


3. Correct speed of speech 


3.03 


2.98 


.93 


.88 


.26 


4. Smoothness of the expansion of the topic 


3.38 


3.36 


.87 


.99 


.05 


5. Ease of speaking 


3.47 


3.91 


1.05 


.96 


1.90 
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Table 3: 



None of Discourse factors, subcategories 

showed any significant differences in Table 3, 
although the main category Discourse factors 
showed significant differences within the 



framework of the 1 1 .main categories. Table 3 
does indicate that Logical combination of sen- 
tences and Flow of ideas are rather highly 
evaluated by the native teachers of English. 



Table 3 




Mean 




SD 




t 


Discourse factors (cohesion and coherence) 


Japanese 


Native 


Japanese 


Native 




1 . Logical combination of sentences 


3.72 


4.05 


.99 


.81 


1.58. 


2. Skills in paragraph development 


3.44 


• 3.16 


1.08 


1.33 


.98 


3. Flow of ideas 


3.69 


4.00 


1.03 


.94 


1.37 



Table 4: 



Although there were significant differences 
in only two of the main categories, Table 4 
shows that significant differences exist within 
some of the subcategories of the remaining 
nine main categories. Some noteworthy ex- 
amples are: Use of grammatically correct word 
order is more highly evaluated by Japanese 
English teachers; the scores for Proper use of 



articles are very low in both groups, though 
there is a slight significant difference; native 
English teachers rate Proper use of tone very 
highly, indicating a dislike for monotonal sen- 
tences; native teachers also put more stress on 
Ability to start and finish a conversation , Abil- 
ity to repair trouble spots in conversations , 
and Ability to manage the utterance act. 



Table 4 




Mean 




SD 




t 




Japanese 


Native 


Japanese 


Native 




Grammatical accuracy 












- Use of grammatically correct word order 


4.00 


3.52 


.86 


.79 


2.34* 


- Length of utterances 


2.60 


3.09 


.84 


.83 


2.57* 


- Correct use of noun-verb agreement 


2.81 


3.41 


1.03 


1.10 


2.39* 


- Ability to use plural forms of nouns 


2.88 


3.36 


.98 


1.04 


2.08* 


- Proper use of articles 


2.50 


2.84 


.76 


.99 


1.63*** 


Pronunciation 












- Proper use of tone 


3.31 


3.86 


.74 


.96 


2.73** 


Level of speaker s confidence 












- Speaker's sureness of phonological accuracy 


3.16 


2.64 


.72 


.84 


2.83** 


Strategic competence 












- Ability to start and finish a conversation 


3.41 


3.98 


.88 


.93 


2.71** 


- Ability to repair trouble spots in conversation 


3.41 


4.11 


.98 


.84 


3.38** 


- Ability to use conversational routines 


3.09 


3.52 


.69 


.79 


2.46* 


Illocutionary competence 












- Ability to manage the utterance act 


3.50 


3.93 


.92 


.87 


2.09* 


- Ability to manage the propositional act 


3.19 


3.66 


.64 


-.86 


2.61* 


- Abilty to manage the illocutionary act 


3.34 


3.66 


.79 


.86 


1.63*** 


* p < .05, 


**p<.01, ***p<. 


10, two-tailed 
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DISCUSSION 

The results presented in Table 1 support the 
first hypothesis that there may be significant 
differences between native and Japanese En- 
glish rating standards in some criteria. How- 
ever, none of the sub-categories of Fluency 
and Discourse factors showed any significant 
differences. This may be because the scores of 
Fluency and Discourse factors in Table 1 are 
not the summed total of individual sub-cat- 
egory scores of Tables 2 and 3. That is, in Table 
1, the raters evaluated Fluency and Discourse 
factors holistically without paying special at- 
tention to the details of each category, and the 
scores of each category were computed and 
analysed within the scope of the 1 1 main cat- 
egories. In contrast, as Tables 2 and 3 show, 
raters gave points to each subcategory and the 
scores were computed and analysed individu- 
ally. Perhaps the raters’ evaluation of the two 
categories, Fluency and Discourse factors, is 
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different, depending on whether they are evalu- 
ating the two categories within the overall 
framework of the 1 1 main categories or from 
separate sub-categories of each. 

Furthermore, while only two main catego- 
ries out of 1 1 showed any significant differ- 
ences, sub-categories of the remaining nine 
main categories did show significant differ- 
ences. There may be two reasons for this dis- 
crepancy. First, as mentioned earlier, the scores 
of the main categories are not the summed total 
of the scores of the sub-categories. Second, the 
raters could evaluate sub-categories in detail, 
while they could only evaluate the main cat- 
egories from the wide, overall perspective. 

The results in Table 5 support the second 
hypothesis that when there are not significant 
differences between the two groups of evalua- 
tors, some criteria will be rated higher by both 
groups of subjects, while both groups will give 
lower points to other criteria. 



Tables 




Mean 




SD 




t 




Japanese 


Native 


Japanese 


Native 




Items which were given lower points 












- Frequency of uncompleted sentences 


2.94 


2.86 


.72 


1.09 


.33 


- How the speaker produces semi-vowels 


3.00 


2.86 


.86 


1.09 


.60 


Items which were given higher points 












- Content 


3.84 


3.82 


1.02 


1.00 


.11 


- Stress 


3.88 


3.89 


.91 


.84 


.06 


- Rhythm 


3.94 


3.89 


.80 


.78 


.28 


- Intonation 


3.97 


3.82 


.74 


.82 


.83 



CONCLUSION 

The profiles of rating standards of both Japa- 
nese English teachers and native English teach- 
ers in evaluating Japanese students’ English 
speaking ability have been described. Partial 
support for the two hypotheses was also found. 
However, the deviation of the scores between 
the main and the sub-categories still exists. 
Therefore, in future research, the construct 



validity and the content validity of both main 
categories and sub-categories must be deter- 
mined. This will be of great value for consid- 
ering rating criteria for tests of English speak- 
ing ability. 
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. Part I 

Directions: When you evaluate Japanese students' 
English speaking ability in class, how much weight 
do you put on each category below? 

Please circle one choice for each category. See the 
example below. 



not 

„ . .. , important important 

Pronunciation (suprasegmental features) 1 2 3 4 5 

30. The naturalness of stress 

31. The naturalness of the intonation 

32. The naturalness of the rhythm 

33. The level of the tone 

34. Proper use of tone (i.e., not monotonic 



not 

important 

EXAMPLE t 

Grammatical accuracy 
Vocabulary use 
Pronunciation (segmental 
features) 



important 

3 4 5 



import an i 
4 5 



i\. c. it you are not sure of the definition of the eleve 
categories below, please refer to the following page 
where you can find some specific items in eac 

cate g°ry- „ ol 

important 

1. Grammatical accuracy \ 2 

2. Vocabulary use 

3. Pronunciation (segmental features) 

4. Pronunciation (suprasegmental features) 

5. Fluency 

6. Discourse (cohesion & coherence factors) 

7. Content 

8 . Level of speaker's confidence 

9. Sociolinguistic competence 

10. Strategic competence 

11. Illocutionary competence 



Part II 

In Part II, each category of Part I will be analysed 
m detail. Please circle one choice for each item 
as in Part I. 



not 

important 

Grammatical accuracy j 

12. Use of grammatically correct word order 

13. Level of sentence complexity 

14. Length of utterances 

15. Correct use of noun-verb agreement 

16. Correct use of tense/aspect form 

17. Ability to use plural forms of nouns 

18. Proper use of articles 

19. Proper use of personal pronouns 

20. Proper use of prepositions 

21. Use of complete sentences 
Vocabulary use 

22. Recognition of nuances 

23. Variety of words 

24. Choice of idioms 
Pronunciation (segmental features) 

25. How the speaker produces vowels 

26. How the speaker produces consonants 

27. How the speaker produces semivowels 

28. How the speaker produces diphthongs 

29. How the speaker produces clusters of sounds 



important 
4 5 



pronunciation) 

Fluency 

35. Proper use of pauses 

36. Frequency of uncompleted sentences 

37. Correct speed of speech 

38. Smoothness of the expansion of the topic 

39. Ease of speaking 

Discourse (cohesion & coherence factors) 

40. Logical combination of sentences 

41. Skills in paragraph development 

42. Flow of ideas 
Content 

43. The creativity or the imaginativeness 
of the speech 

Level of speakers confidence 

44. Speaker's certainty of the grammatical 
accuracy 

45. Speaker's sureness of the phonological 
accuracy 

46. Speaker's confidence in the choice of 
words 

Sociolinguistic competence (difference in 
register or difference in variation in 
language use) 

47. Ability to handle the field of discourse 
(the appropriate language use in the 
language context) 

48. Ability to handle the mode of discourse 
(the ability to attest to the differences 
between written and spoken variation in 
language use) 

49. Ability to handle the tenor of discourse 
(the use of appropriate style among the 
participants in certain language use contexts) 
Strategic competence (in the case of 
interview or role-play) 

50. Ability to manage turn-taking (taking a 
turn, holding a turn and relinquishing a turn) 

51. Ability to start and finish a conversation 

52. Ability to initiate and respond to remarks 
on a broad range of topics 

53. Ability to develop and continue speaking 
on topics 

54. Ability to repair trouble spots in conver- 
sation (communication breakdown or comp- 
rehension problems) 

55. Ability to use conversational fillers and 
small talk 

56. Ability to use conversational routines 
Illocutionary competence (in the case of 
interview or role-play) 

57. Ability to manage the utterance act (the 
utterance act: the act of saying something 

58. Ability to manage the propositional act 
(a propositional act: referring to something, 
or expressing a predication about something) 

59. Ability to manage the illocutionary act 
[the illocutionary act: the function (e g., 
assertion, warning, request) performed in 
saying something] 



BEST COPy AVAILABLE 



7 



English Today 

The International Review of the English Language 



About the Journal... 

English Today is for everyone 
concerned with or fascinated by the 
English language. It covers all aspects 
of the language, including its uses and 
abuses, its international variations, its 
history, its literature and linguistics, 
usages and neologisms. Special 
articles and regular features keep you 
up to date with current opinion and 
recent developments. English Today 
is for teachers and students of English 
as a first or second language; for 
writers, broadcasters, journalists, 
linguists; and anyone who is 
concerned about the English 
Language. 

Published by Cambridge University 
Press, this quarterly journal provides 
a unique forum for both foreign and 
native users of English. It successfully 
combines academic rigour with 
popular appeal in a field where 
controversy abounds. 



Coverage 

EnglishTodaybringsyou scholarship, 
comment, information and humour 
from around the world. 

English Today draws on a wide range 
of contributors including academics, 
teachers, publishers and journalists - 
all professional users of the English 
language. 

English Today thrives on the flood of 
correspondence it receives from its 
readers. A broad cross-section of the 
letters is published in each issue, 
demonstrating the enthusiasm and 
erudition of professional and amateur 
alike. 

Subcription Information 

English Today, Volume8(ET29-32); 
January, April, July and October in 
1992: £41 for institutions; £19 for 
individuals; £16 for students and the 
retired. 



□ Please send me further information on English Today 

Name 

Address 



Sendto: Journals Marketing Department, Cambridge University Press, FREEPOST*, 
The Edinburgh Building, Cambridge CB2 1BR, England TEL: (0223) 325806 or 
FAX (0223) 315052 (* no postage stamp necessary if posted in UK) 

In US & Canada, write to Cambridge University Press, 

40 West 20th Street, New York, NY 10011-4211, USA 



iCambru(ffe 
a Journals 



8 





iV .‘vi .‘.L; I 









® 

U.S. Department of Education 

Office of Educational Research and Improvement (OERI) 

National Library of Education (NLE) 

Educational Resources Information Center (ERIC) 

REPRODUCTION RELEASE 

(Specific Document) 



I. DOCUMENT IDENTIFICATION: 



Title: Pl'f-f? r-W(£s in A/// V /[/ 7^ c/u/^s y 

^iiAsLiA* 9 fry} 




Author(s): Vm ,V A^Cf Md /'* t 


Corporate Source: 




Publication Date: 


Cro^<? Curr-eni y 




h/.'r) 



II. REPRODUCTION RELEASE: 



In order to disseminate as widely as possible timely and significant materials of interest to the educational community, documents announced in the 
monthly abstract journal of the ERIC system, Resources in Education (RIE), are usually made available to users in microfiche, reproduced paper copy, 
and electronic media, and sold through the ERIC Document Reproduction Service (EDRS). Credit is given to the source of each document, and, if 
reproduction release is granted, one of the following notices is affixed to the document. 

If permission is granted to reproduce and disseminate the identified document, please CHECK ONE of the following three options and sign at the bottom 
of the page. 





The sample sticker shown below will be 
affixed to all Level 1 documents 



The sample sticker shown below will be 
affixed to all Level 2A documents 



The sample sticker shown below will be 
affixed to all Level 2B documents 



PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL HAS 
BEEN GRANTED BY 

* 




PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL IN 
MICROFICHE, AND IN ELECTRONIC MEDIA 
FOR ERIC COLLECTION SUBSCRIBERS ONLY, 
HAS BEEN GRANTED BY 

A® 




PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL IN 
MICROFICHE ONLY HAS BEEN GRANTED BY 

$ 


cf* 




cA 




Ay 


TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 

1 




TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 

2A 




J 

TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 

2B 



Level 1 



Level 2A 



Level 2B 



[ 7 ] 



i 



t 




Check here for Level 1 release, permitting reproduction 
and dissemination in microfiche or other ERIC archival 
media (e.g., electronic) and paper copy. 



Check here for Level 2A release, permitting reproduction 
and dissemination In microfiche and In electronic media 
for ERIC archival collection subscribers only 



Check here for Level 2B release, permitting 
reproduction and dissemination In microfiche only 



Documents will be processed as Indlceted provided reproduction quality permits. 

If permission to reproduce Is granted, but no box is checked, documents will be processed at Level 1 . 



Sign 

here,-* 

please 



O 

ERIC 



/ hereby grant to the Educational Resources Infonriation Center (ERIC) nonexclusive penriission to reproduce and disseminate this document 
as indicated above. Reproduction from the ERIC microfiche or electronic media by persons other than ERIC employees and its system 
contractors requires penriission from the copyright holder. Exception is made for non-profit reproduction by libraries and other service agencies 
to satisfy infonnation needs of educators in response to discrete inquiries. 


Siflna,ure 


Printed Name/PositiorVTitle: . _ _ f 

Yu<T I A/dM/V Ufa 


Organizalion/^Sress: £ ~ 

1-7-} 7 cAo ' 


Telephone: 




E-Mail Address: s) / 


Da,e: ry^/^ 


prv i 


/s 1 yf 


(over) 






